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Let Cognitive Radios Imitate: Imitation-based 
Spectrum Access for Cognitive Radio Networks 

Stefano lellamo Lin Chen Marceau Coupechoux 

Abstract 

In this paper, we tackle the problem of opportunistic spectrum access in large-scale cognitive radio networks, 
where the unlicensed Secondary Users (SU) access the frequency channels partially occupied by the licensed 
Primary Users (PU). Each channel is characterized by an availability probability unknown to the SUs. We apply 
evolutionary game theory to model the spectrum access problem and develop distributed spectrum access policies 
based on imitation, a behavior rule widely applied in human societies consisting of imitating successful behavior. 
We first develop two imitation-based spectrum access policies based on the basic Proportional Imitation (PI) rule 
and the more advanced Double Imitation (DI) rule given that a SU can imitate any other SUs. We then adapt the 
proposed policies to a more practical scenario where a SU can only imitate the other SUs operating on the same 
channel. A systematic theoretical analysis is presented for both scenarios on the induced imitation dynamics and the 
convergence properties of the proposed policies to an imitation-stable equilibrium, which is also the e-optimum of 
the system. Simple, natural and incentive-compatible, the proposed imitation-based spectrum access policies can be 
implemented distributedly based on solely local interactions and thus is especially suited in decentralized adaptive 
learning environments as cognitive radio networks. 

I. Introduction 

Cognitive radio with its capability to flexibly configure its transmission parameters, has emerged 
in recent years as a promising paradigm to enable more efficient spectrum utilization. Spectrum access 
models in cognitive radio networks can be classified into three categories, namely exclusive use (or operator 
sharing), commons and shared use of primary licensed spectrum |[2l. In the last model, unlicensed secondary 
users (SU) are allowed to access the spectrum of licensed primary users (PU) in an opportunistic way. In 
this case, a well-designed spectrum access mechanism is crucial to achieve efficient spectrum usage. 

In this paper, we focus on the generic model of cognitive networks consisting of multiple frequency 
channels, each characterized by a channel availability probability determined by the activity of PUs on it. 
In such model, from the individual SU's perspective, a challenging problem is to compete (or coordinate) 
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with other SUs in order to opportunistically access the unused spectrum of PUs to maximize its own 
payoff (e.g., throughput); at the system level, a crucial research issue is to design efficient spectrum access 
protocols achieving optimal spectrum usage. 

We tackle the spectrum access problem in large-scale cognitive radio networks from an evolutionary 
game theoretic angle. We formulate the spectrum access problem as a non-cooperative game and develop 
distributed spectrum access policies based on imitation, a behavior rule widely applied in human societies 
consisting of imitating successful behavior. We establish the convergence of the proposed policies to an 
imitation-stable equilibrium which is also the e-optimum of the system. Simple, natural and incentive- 
compatible mechanism, the proposed spectrum access policies can be implemented distributedly based on 
solely local interactions and thus is especially suited in decentralized adaptive learning environments as 
cognitive radio networks. 

The motivation of applying evolutionary game theory and imitation-based strategy in the study of the 
spectrum access problem is tri-fold. 

• First, evolutionary game theory is a powerful tool to study the interaction among players and the 
system dynamic in terms of population. Stemmed from classic game theory and Darwin's evolution 
theory, it can explicitly capture the fundamental relationship among competition, cooperation and 
communication, three crucial elements in the design of any spectrum access protocols in cognitive 
radio networks. 

• Second, compared with replicator dynamic, the most explored evolutionary model which mimics the 
effect of natural selection, imitation dynamic captures the spreading of successful strategies through 
imitation rather than inheritance, which is more adapted in games played by autonomous decision 
makers as in our case. 

• Third, evolutionary game theory, especially imitation dynamic which relies solely on local interactions, 
provides a theoretic tool for the design of distributed channel access protocols based on local 
information which is particularly suited in decentralized environments as cognitive radio networks. 

In our analysis, we start by developing the imitation-based spectrum access policies where a SU can 
imitate any other SUs. More specifically, we develop two spectrum access policies based on the following 
two imitation rules: 

• the Proportional Imitation (PI) rule where a SU can sample one other SU; 

• the more advanced adjusted proportional imitation rule with double sampling (Double Imitation, DI) 
where a SU can sample two other SUs. 

Under both imitation rules, each SU strives to improve its individual payoff by imitating other SUs with 
higher payoff. We then adapt the proposed spectrum access policies to a more practical scenario where 
a SU can only imitate the other SUs operating on the same channel. A systematic theoretical analysis 
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is presented for both scenarios on the induced imitation dynamics and the convergence properties of the 
proposed policies to an imitation-stable equilibrium, which is also the e-optimum of the system. 

The key contribution of our work in this paper lies in the systematical application of the natural imitation 
behavior to address the spectrum access problem in cognitive radio networks, the design of a distributed 
imitation-based channel access policy, and the theoretic analysis on the induced imitation dynamic and 
the convergence to an efficient and stable system equilibrium. 

The rest of the paper is structured as follows. Section |Il] presents the system model followed by the 
formulation of the spectrum access game. Section UlI] describes the proposed imitation-based spectrum 
access policies in the scenario where a SU can imitate any other SUs. In Section |IVl we adapt the 
proposed policies to the scenario where a SU can only imitate the other SUs operating on the same 
channel. Section |V] presents simulation results on the performance of the proposed policies. Section |VI] 
discusses related work in the literature. Section IVIII concludes the paper. 

II. System Model and Spectrum Access Game Formulation 

In this section, we present the system model of our work with the notations used, followed by the 
game formulation of the spectrum access problem, which serves as the basis of the analysis in subsequent 
sections. 



A. System Model 

We consider a primary network consisting of a set C of C frequency channels, each with bandwidth i^. 
The users in the primary network are operated in a synchronous time-slotted fashion. A set J\f of N SUs 
tries to opportunistically access the channels when they are left free by PUs. Let Zi{k) be the random 
variable equal to 1 when of channel i is unoccupied by any PU at slot k and otherwise. We assume that 
the process {Zi{k)} is stationary and independent for each i and k. We also assume that at each time slot, 
channel i is free with probability fii, i.e., E[Zj(A;)] = /ij. The channel availability probabilities ^ = {^i} 
are a priori not known by SUs. We assume perfect sensing at the SUs, i.e., any transmission of any PU 
on a channel is perfectly sensed by SUs sensing that channel and thus no collision occurs between PUs 
and SUs. 

In our work, each SU j is modelled as a rational decision maker, striking to maximize the throughput 
it can achieve, denoted as Tj, which can be expressed as a function of /i^ and Us^, where sj denotes the 
channel which j chooses, Ug. denotes the number of SUs on channel Sj. More formally, the expected 
value of Tj can be written as: 

'Our analysis can be extended to study the heterogeneous case with different channel capacities. 
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In order to perform a closed-form analysis, we focus on the scenario where the channel capacity is evenly 
shared among all SUs on the channel when it is free, i.e.. 



It should be noted that f{fisj,ns-) depends on the MAC protocol implemented at the cognitive users. 
Beside the evenly shared model considered in this paper, several other models are also largely applied in 
practice such as the CSMA-based random access model. Our work in this paper can be adapted in those 
cases by defining appropriate function /. 

B. Spectrum Access Game Formulation 

To study the interactions among autonomous selfish SUs and to derive distributed channel access policies, 
we formulate the channel selection problem as a spectrum access game where the players are the SUs. 
Each player j stays on a channel i to opportunistically exploit the unused spectrum of PUs to maximize 
its expected throughput. The game is defined formally as follows: 

Definition 1. The spectrum access game G is a 3 -tuple (Af, C, {Uj}), where Af is the player set, C is the 
strategy set of each player Each player j chooses its strategy sj E C to maximize its normalized utility 
function Uj defined as 



The solution of the spectrum access game G is characterized by a Nash Equilibrium (NE) [Q, a strategy 
profile from which no player has incentive to deviate unilaterally. Using the related theory on congestion 
games, we can establish the existence and the uniqueness of the NE in the spectrum access game G for 
the asymptotic case (A^ — > oo) in the following theorem. 

Theorem 1. In the asymptotic case where N is large, G admits a unique NE. At the NE, there are x*N 
SUs staying with channel i, where x* = ^=r^ — . 

Proof: Given the form of SUs' utility function, it follows from in that the spectrum access game 
is a congestion game. Moreover, in the asymptotic case approximating the game G by a game with a 
continuous set of users, denote x = {xi,i E C} where Xi denotes the proportion of SUs choosing channel 
i, we can write the potential function of the congestion game as follows: 



where eo > is a small constant introduced to avoid the non-integral point of fii/t at 0. We can verify 
that for a SU j staying on channel i, it holds that : 



U, = E[T,]/B = fisJn 




dP{x) 
dxi 



¥.[Uj{iiuXiN)]. 
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To derive the NE of G, we seek the maximum of P{x). To this end, we develop P{x) as 

^(^) = X^^'^^^Sa;* -logeo). 
To find the maximum of P(x), we solve the following optimization problem 



max P{x) s.t. 7 Xj = 1 and Xi > 0,Vi G C, 



X 

iec 



which has a unique solution because the KKT conditions are necessary and sufficient as P{x) is concave 
and the constraint is linear. After some straightforward algebraic operations, we can find the unique 
maximum x* = {x*} as follows: 

The maximum x* is also the unique NE of G. ■ 
We can observe two desirable properties of the unique NE derived in Theorem [1} 

• the NE is optimal from the system perspective as the total throughput of the network achieves its 
optimum at the NE; 

• the NE ensures that the spectrum resource is shared fairly among SUs. 

One critical challenge in the analyzed spectrum access game is the design of distributed spectrum access 
strategies for rational SUs to converge to the NE without the a priori knowledge of fi. In response to 
this challenge, we develop in the sequel sections of this paper an efficient spectrum access policy. Our 
proposed policy can be implemented distributedly based on solely local interactions without any knowledge 
on the channel statistics and thus is especially suited in decentralized adaptive learning environments as 
cognitive radio networks. In terms of performance, we demonstrate both analytically and numerically that 
the proposed channel access policy converges to the e-NEn of G which is also the e-optimum of the system. 

III. Imitation-based spectrum access policies 

The spectrum access policy we develop is based on imitation. As a behavior rule widely observed in 
human societies, imitation captures the behavior of a rational player that mimics the actions of other 
players with higher payoff in order to improve its own payoff. The induced imitation dynamic models the 
spreading of successful strategies under imitation Q. In this section, we focus on the scenario where a SU 
can imitate any other SUs and develop two spectrum access policies based on the proportional imitation 
rule and the double imitation rule. We analyze the induced dynamic of the imitation process and show 
the convergence of the proposed policy to the e-NE of G. In the next section, we extend our efforts to a 
more practical scenario where a SU can only imitate the other SUs operating on the same channel and 
develop an adapted imitation-based spectrum access policy in the new context. 

strategy profile is an e-NE if no player can gain more than e in payoff by unilaterally deviating from his strategy. 
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A. Spectrum Access Policy Based on Proportional Imitation 

Algorithm [T] presents our proposed spectrum access policy based on the proportional imitation rule, 
termed as PISAP. The core idea is: at each iteration, each SU randomly selects another SU in the network; 
if the payoff of the selected SU is higher than its own payoff, the SU imitates the strategy of the selected 
SU at the next iteration with a probability proportional to the payoff difference, coefficiented by the 
imitation factor crO 

We first study the dynamic induced by PISAP by setting ejj = 0. It is shown in |l6l that in the asymptotic 
case, the proportional imitation rule in Algorithm [T] generates a population dynamic described by the 
following set of differential equations: 

x^{t)=axiit)[7r,{t)-7r{t)] ieC, (1) 

where tt^ denotes the expected payoff of the SUs on channel i,n = J2iec denotes the expected payoff 
of all SUs in the network. Injecting tTj = fii/{xiN) into the differential equations, ([B becomes: 



Xijt) _ iM _ (.\\^ ta 

a -N ""'^^'l^N- 

i&c 



This equation can be easily solved as: 



x.(t) = K.e-(^-^^V + ^^^, (2) 

where the constant Ki = Xj(0) — -^^r^ — . 

As the first result of this section, the following theorem states the convergence of the dynamic to the 
NE of the spectrum access game G. 

Theorem 2. The imitation dynamic induced by PISAP converges exponentially to an evolutionary equi- 
librium which is also the NE of G. 

Proof: The theorem follows straightforwardly from Q and Theorem [H ■ 
As an illustrative example. Figure [U (obtained with [!7|) shows the convergence of the imitation dynamic 
of PISAP to the NE of G for a cognitive network of 2 channels and 50 SUs. 

We then study the convergence of PISAP in the general case with e^/ > 0. Specifically, we define the 
imitation- stable equilibrium as a state where no further imitations can be conducted based on the imitation 
policy |[8l. The following theorem analyzes the convergence of PISAP with respect to this concept. 

Theorem 3. PISAP converges to an imitation- stable equilibrium in expected 0( — — — ) iterations where 
jjimin — minjgc /^i- The converged equilibrium is an e-NE of G with e = 2eu- 

Proof: We provide the sketch of the proof here. The detailed proof is provided in the Appendix. 
We first prove the convergence of PISAP to an imitation-stable equilibrium. Define i"^^'"" — argmax^g^ T^iif) 

'One way of setting a is to set a — ^/{oj — a), wfiere uj and a are two exogenous parameters such that Uj £ [a, ttj], Vj £ C. 
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and z™" = argmiiijgc 7ri(t), we show that for each iteration t, if 7iim.ax(t) — 7rjmin(t) > e;/, then at least 
one of the following holds 

TTj^n (t + 1) - TTi^.n (t) ~ O ^'^'^r'' ) 
7rima.{t) - nima.{t + 1) ~ Q ( Mmo^crct/ ^ 

I.e., if the difference between the highest expected individual payoff TTimax{t) and the worst one 7rjm»n(t) is 
larger than eu, we can at least increase 7iirmn(t) by 0( ^"^i^r^ ) or decrease 7rjma.(t) by O(^^^^^fl^). 
It follows that after a finite number of iterations (the exact form is deduced in the detailed proof), 
PISAP converges to a state where 7rjma:r(t) — 7rjmm(t) < eu, which is imitation- stable. We then show 
by contradiction that the converged imitation- stable equilibrium is an e-Nash of G with e = 2eu- ■ 
Note that the convergence delay ^L^, ) derived in Theorem [3] consists of the upper bound and 
through the simulations we conduct, we observe that the convergence is achieved in a much shorter delay. 



B. Spectrum Access Policy Based on Double Imitation 

In this subsection, we turn to a more advanced imitation rule, the double imitation rule [9] and propose 
the Dl-based spectrum access policy, termed as DISAP. Under DISAP, each SU randomly samples two 
SUs and imitates them with a certain probability determined by the utility difference. The spectrum access 
policy based on the double imitation is detailed in Algorithm [21 in which each SUs randomly samples 
two other SUs ji and j2 (without loss of generality, assume that ji and j2 operate on channel ii and Z2 
respectively, with corresponding utilities Uj-^ < Uj^) and updates the probabilities of switching to channels 
ii and i2, denoted as pj-^ and pj^ respectively. 

The double imitation rule generates an aggregate monotone dynamic Bll, [fTOl . which is defined as 
follows: _ 

(vTi - 7f) Vz G C (3) 



Xi 



u — a 



UJ — IT 

oj — a 



Injecting tt^ = Hi/{xiN) into the differential equations, we have: 

an ( (jj — 7f\ (jTf / (jj — 7f\ 

Xi = 1 + 1 + Xi, 

UJ — a \ Lo — a J io — a \ lo — a / 

whose solution is 

= Ke-^i'^^y + (4) 

where vf = J2iec l^il^ ^ — Xi{0) — ^ ^' . In the studied scenario, a and cu are the lower and upper 
bound of the SUs' utility, which are and 1, respectively. 

The following theorem stating the major result in this subsection follows immediately. 

Theorem 4. DISAP converges exponentially to the NE of the spectrum access game G. 

Compared with the proportional imitation rule, which produces the replicator dynamic (Eq. O), the 
adjusted proportional imitation rule induces the aggregate monotone dynamic (Eq. ([3])) that converges to 
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the NE at a higher rate. In Fig. [T] is shown the phase plane for replicator dynamics and aggregate monotone 
dynamics. As proven in Theorem [T] for large N there exists only one attractor (NE), which is the crossing 
point of the two nulclines (dashed lines). 

We then study the convergence to an imitation- stable equilibrium of DISAP in the general case with 
6(7 > in the following theorem. 

Theorem 5. DISAP converges to an imitation- stable equilibrium in expected ^\ejj ) iterations where 
jjimin — minjgc l^i- The converged equilibrium is an e-NE of G with e = 2eu- 

Proof: The proof follows the same analysis as that of Theorem [3j The detail is provided in the 
Appendix. ■ 

C. Discussion 

As desirable properties, the proposed imitation-based spectrum access policies (both PISAP and DISAP) 
are stateless, incentive-compatible for selfish autonomous SUs and requires no central computational unit. 
The spectrum assignment is achieved by local interactions among autonomous SUs and the e-optimum 
of the system is achieved when the algorithm converges, which is achieved in polynomial time. The 
autonomous behavior and decentralized implementation make the proposed policies especially suitable for 
large scale cognitive radio networks. The imitation factor a controls the tradeoff between the convergence 
speed and the channel switching frequency in that larger a represents more aggressiveness in imitation 
and thus leads to fast convergence, at the price of more frequent channel switching for the SUs which 
may consist of significant cost for today's wireless devices in terms of delay, packet loss and protocol 
overhead. The imitation threshold eu, on the other hand, can be tuned to balance between the convergence 
speed and the optimality of the converged equilibrium. 

IV. Imitation on the same channel 

Up to now, we have studied the imitation-based channel access policy where a SU can imitate any other 
SU whatever the channel the latter stays in. This approach implicitly assumes that a SU can interact with 
SUs on different channels, which may not be realistic in some cases or pose additional system overhead 
(e.g., sensing a different channel). In this section, we focus on a more practical scenario, where a SU 
only imitates the SUs on the same channel and the imitation is based on the payoff difference of the 
precedent iteration. In the considered scenario, a SU only needs to locally interact with the SUs on the 
same channel (e.g., exchange payoff of the precedent iteration, which can be piggybacked with the data 
packets transmitted on the channel). 

In the sequel analysis, we first study the induced imitation dynamic and the convergence of the proposed 
spectrum access policies PISAP and DISAP subject to channel constraint on imitation. 
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A. Imitation Dynamic and Convergence 

In this subsection, we first derive in Theorem [6] the dynamic for a generic imitation rule F with 
large population. We then derive in Lemma [H Theorem |7] and Theorem [8] the dynamic of the proposed 
proportional imitation policy PISAP and its convergence under the channel constraint. The counterpart 
analysis for the double imitation policy DISAP is explored in Lemma [2l Theorem |9] and Theorem [lOl 

We start by introducing the notations used in our analysis. At an iteration, we label all SUs performing 
strategy i (channel i in our case) as SUs of type i and we refer to the SUs on Sj as neighbors of SU j. 
We denote n[{t) the number of SUs on channel i at iteration t and operating on channel / at t — 1. It 

holds that X]jgc'^i(^) = ^ii'^) J2i£c''^i(^) ^ ''^'(^ ~ -'-)• ^ given state s{t) = {sj{t),j G C} at 
iteration t and a finite population of size N, we denote pi{t) = ni{t)/N the proportion of SUs of type i 
and p[{t) = n\{t) /N the proportion of SUs migrating from channel / to i. We use x instead of p to denote 
these proportions in asymptotic case. It holds that p ^ x when N +oo. 

In our study, a generic imitation rule under the channel constraint is termed as F. In the case of the 
proportional imitation rule (c.f. PISAP), F is characterized by the probability set {F^^} where Fj^ denotes 
the probability that a SU choosing strategy j at the precedent iteration imitates another SU choosing strategy 
k at the precedent iteration and then switches to channel i at next iteration after imitation. Instead, by 
applying the double imitation rule (c.f. DISAP), we can characterize F by the probability set {Fji^ki}} 
where |^ ;| denotes the probability that a SU choosing strategy j at the precedent iteration imitates two 
neighbors choosing respectively strategy k and strategy / at the precedent iteration and then switches to 
channel i at next iteration after imitation. In both cases the only way to switch to a channel i is to imitate 
a SU that was on channel i. That means Fj^,^ = 0,\/k^i (PISAP) and Fj^^^ ;j = 0,\/kJ^i (DISAP). 
At the initialization phase (iteration and 1), each SU randomly chooses its strategy. After that, the system 
state at iteration t + 1, denoted as p(t + 1) (x(t + 1) in the asymptotic case), depends on the states at 
iteration t and t — 1 . 

Theorem 6. For any imitation rule F, if the imitation among SUs of the same type occurs randomly 
and independently, then V(5 > 0, e > and any initial state {xj(0)}, {xj(l)}, there exists Nq E N 
such that if N > Nq, Wi G C, the event \pi(t) — Xi(t)\ > 6 occurs with probability less than e, where 
Pi{0) = Xj(0) = Xi{0), Pi{l) = Xi{l) = Xi{l). In the case of proportional imitation policy it holds that 



Proof: The proof consists of first showing the theorem holds for iteration t = 2 and then proving the 




Differently, the double imitation policy yields: 
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case t > 3 by induction. The detail is in Appendix. ■ 
Theorem [6] is an important result on the short run adjustments of large populations under any generic 

imitation rule F: the probability that the behavior of a large population differs from the one of an infinite 

population is arbitrarily small when N is sufficiently large. In what follows, we study the convergence of 

PISAP and DISAP under the channel constraint. 

(1) Spectrum access policy PISAP under channel constraint 

We now focus on PISAP under channel constraint and derive the induced imitation dynamic by setting 
= in the following analysis. 

Lemma 1. On the proportional imitation policy PISAP under channel constraint, it holds that 

x\it)x^At) . 

^K^ + l)= E %[t) ^^^' "^^^J^C. (5) 



Lk&C 



Proof: The proof is straightforward from the analysis in the proof of Theorem [61 ■ 

Theorem 7. The proportional imitation policy PISAP under channel constraint generates the following 
dynamic in the asymptotic case 

(t)x'' (t) 

x,{t + 1) = Xi{t - 1) + aTii{t - l)xi{t - 1) - a E Tii{t - 1) (6) 
where 'n'i{t) denotes the expected payoff of an individual SU on channel i at iteration t. 

Proof: It can be shown that the proportional imitation policy PISAP under channel constraint is both 
imitatingj and improving^ Apply the analysis in (Si (Eq. (10)) to our case, we can characterize {Fl,^} for 
the proportional imitation rule under channel constraint as: 

ky^i 

Fl + a[TT^{t-l)-ni{t-l)] k = i 
In other words, the only possibility to switch to channel i is to imitate a SU that was on channel i; the 
switching probability is proportional to the payoff difference. Noticing that ^ji^) = 1, ® 

can be written as follows: 

^xUt)xi(t) 

Xj{l) 



lec 

Injecting Y^jX^j{t) = a:j (t — 1 ) , x'- (t) / Xj (t) = 1 and x j (t + 1 ) = (t + 1 ) into the above formula, 

we can obtain which concludes the proof. ■ 

■*A behavior rule is imitating if tlie switching actions of the rule occur by imitating the sampled individual. 
'a behavior rule is improving if and only the expected payoff of an individual increases after imitation. 
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We observe via extensive numerical experiments that © always converges to an evolutionary equilib- 
rium. To get more in-depth insight on the dynamic we notice that under the following approximation: 

$^vrKt-l)-^^7r(t-l), (7) 
lec ' 

where vf (t — 1) is the average individual payoff for the whole system at iteration t — \, noticing (t) = 

Xi{t — 1), ^ can be written as: 

Xi{t + 1) = x,{t - 1) + ax,{t - l)[iT,{t - 1) - 7r(t - 1)]. (8) 

Note that the approximation (|7]) states that in any channel j at iteration t, the proportions of SUs coming 
from any channel / are representative of the whole population. 

Under the approximation ([7]), given the initial state {xi(0)}, {xi{l)}, we can decompose dH]) into the 
following two independent discrete-time replicator dynamics: 

Xiiu) = Xiiu — 1) + axiiu — l)[7rj('u — 1) — t^{u — 1)] 

Xi{v) = Xi{v - 1) + (TXi{v - l)[Hi{v - 1) - 7l{v - 1)] 

where u = 2t, v = 2t + 1. The two equations in Q illustrate the underlying system dynamic hinged 
behind the proportional imitation policy under channel constraint under the approximation ([7]): it can be 
decomposed into two independent delayed replicator dynamics that alternatively occur at the odd and even 
iterations, respectively. The following theorem establishes the convergence of dH) to a unique fixed point 
which is also the NE of the spectrum access game G. 

Theorem 8. Starting from any initial point, the system described by ^ converges to a unique fixed point 
which is also the NE of the spectrum access game G. 

Proof: The proof, of which the detail is provided in the Appendix, consists of showing that the 
mapping described by (l9l) is a contraction mapping. ■ 

As an illustrative example. Figure |2] shows that the double replicator dynamic provides an accurate 
approximation of the system dynamic induced by PISAP under channel constraint. 

Furthermore, performing the same analysis as that of Theorem [3l we can establish the same convergence 
property on the imitation algorithm under channel constraint under the approximation (|7]) for the general 
case with e^/ > 0. 

(2) Spectrum access policy DISAP under channel constraint 

We then focus on DISAP under channel constraint and derive the induced imitation dynamic. 

Lemma 2. On the double imitation policy DISAP under channel constraint, it holds that 

xl{t + l)=Y. [j'^^lf ^ FliK^} Vz,, G C. (10) 
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Proof: The proof is straightforward from the analysis in the proof of Theorem [6l ■ 

Theorem 9. The double imitation policy DISAP under channel constraint generates the following dynamic 
in the asymptotic case 



Xiit + 1) = Xiit - 1) + 2x,{t - l)7ri(t - 1) + 



-x5(t)7r,(t-l)5^-^7r,(t-l) (11) 
k ^ 

where iiiit) denotes the expected payoff of an individual SU on channel i at iteration t. 

Proof: If the rule F is unbiased, it follows from (9\ that F is also improving and globally efficient. 
We can than characterize {Fl^} as in @| (Theorem 1): 

Fi,{k,i} = Pi,{i,k} + Fl{k,i} - Fk,{i,i} ~ 2^(^fc)(^' ~ ~ 2^*^^''**^^'' ~ 
In other words, the only possibility to switch to channel i is to imitate a SU that was on channel i. 
Setting uj = I and a = 0, so that a{y) = 2 — y, and noticing that Yli k^''j(^)^ji^)-^k {i ij/^ji'^)'^ = 1, © 
can be written as follows: 

x^(t)7rfc(t-l)\ / , x^7rfc(t-l) 



Injecting = Xi{t — 1), Xlz = 1 ^^'^ Xi{t + 1) = + into the above formula, 

we can obtain (fTTI) . which concludes the proof. ■ 

We observe via extensive numerical experiments that (fTTI) always converges to an evolutionary equilib- 
rium and, as shown in Fig, [6l also features a smoother and faster convergence trend with respect to the 
proportional imitation dynamic (Eq. 

By performing the same approximation as in (|7]), (fTTI) can be written as: 

Xi{t + 1) = Xi{t - 1) + Xi{t - 1)(2 - n{t - l)){7ii{t - 1) - n{t - 1)). (12) 

Under the approximation ([7]), given the initial state {xi{0)}, {xj(l)}, we can decompose (fT2l) into the 
following two independent discrete-time aggregate monotone dynamics: 

Xi{u) = Xi{u — 1) + Xi{u — 1)[2 — 7f(n — 1)] ■ [7ii{u — 1) — 7f(n — 1)] 

(13) 

Xi{v) = Xi{v - 1) + Xi{v - 1)[2 - tt{v - 1)] ■ [miv - 1) - tt{v - 1)] 
where u = 2t, v = 2t + l. The above two equations illustrate the underlying system dynamic hinged behind 
the double imitation policy under channel constraint under the approximation (|7]): it can be decomposed 
into two independent delayed aggregate monotone dynamics that alternatively occur at the odd and even 

*A behavioural rule is unbiased if it does not depend on the labelling of actions. 

'a behavioural rule is globally efficient if, for any Multi-armed bandit, all individuals use a best action in the long run, provided that 
initially each action is present. 
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iterations, respectively. The following theorem establishes the convergence of (fT3]) to a unique fixed point 
which is also the NE of the spectrum access game G. The proof follows exactly the same analysis as that 
of Theorem [8l 

Theorem 10. Starting from any initial point, the system described by (fT3]) converges to a unique fixed 
point which is also the NE of the spectrum access game G. 

As an illustrative example, Figure [3] shows that the double aggregate dynamic provides an accurate 
approximation of the system dynamic induced by the double imitation under channel constraint. 

B. Imitation-Based Channel Access Policy under Channel Constraint 

In this subsection, based on the theoretic results derived previously, we develop a fully distributed 
channel access policy for the general case with finite population based on the imitation rule among SUs 
on the same channel (i.e. neighbors). The proposed policy, detailed in Algorithm [3l is suitable both for 
proportional and double imitation. Run at each SU j and at each iteration, it consists of: 

• sampling randomly one (proportional imitation) or two (double imitation) neighbors; 

• comparing the payoff achieved at the previous iteration t — \ with that of the neighbor(s) selected 
for imitation; 

• performing channel migration with the probability dictated by the applied imitation rule. 
Algorithm |3] is evaluated by extensive simulations in next section. 

V. Performance Evaluation 

In this section we conduct extensive simulations to evaluate the performance of the proposed imitation- 
based channel access policy (PISAP and DISAP) in both scenarios with and without channel constraints 
and demonstrate some intrinsic properties of the policy which are not explicitly addressed in the analytical 
part of the paper. 

A. Simulation Settings 

We simulate a cognitive radio network of = 50 SUs and C = 3 channels, on which PUs has different 
activity rates on different channels, leading to different channel availability probabilities characterized by 
ji = [0.3,0.5,0.8]. We assume the iteration time to be long enough so that the SUs, regardless of the 
occupied channel, can evaluate their payoff without errors. 

B. System Dynamics 

We first study the system dynamic induced by PISAP and DISAP in the scenarios with and without 
channel constraint. 
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As illustrated by Fig. |4] and Fig. [51 both ^ and (dB, which reflect PISAP and DISAP behaviour in 
the asymptotic case with channel constraint, produce trajectories that converge in a faster but less smooth 
manner if compared with their respective unconstrained dynamics. This can be interpreted by the overlap 
of two replicator/aggregate monotone dynamics at odd and even instants, as explained in section |IVl 
In Fig. [6] the trends of ^ and (fTTI) are compared. We observe that, in the asymptotic case, DISAP 
outperforms PISAP as it is characterized by less pronounced wavelets and a faster convergence. However, 
all the displayed dynamics correctly converge to an evolutionary equilibrium. It is easy to check that the 
converged equilibrium is also the NE of G and the system optimum, which confirms our theoretic analysis. 

C. Convergence to Imitation- stable Equilibrium 

We now study the convergence of PISAP and DISAP with and without channel constraint for a finite 
number of users {N = 50). Fig. |7] and Fig. [8] show the number of SUs per channel during the convergence 
phase for one realization of our algorithms without channel constraint. We notice that in both cases 
convergence is rapidly achieved after few iterations, and that the channels with higher availabilities are 
chosen by more individuals. This can be easily verified (Fig. |7] for PISAP and Fig. [8] for DISAP) by 
observing that after convergence the major part of population settles permanently in channel 3, i.e. the 
channel that less frequently hosts PU's transmissions. 

Fig- m and Fig. [TOlshow that, starting from the same initial conditions, DISAP reaches an imitation- stable 
equilibrium more rapidly than PISAP, at the price of a higher algorithmic complexity and a substantial 
increase in information exchanges amongst the SUs due to double imitation. Note that the small deviation 
of the trajectories at some iterations in the figures from the converged curve is due to the probabilistic 
nature of the users' strategy and has only very limited impact on the system as a whole. 

Fig. \TT\ and Fig. [12] show instead a realization of our algorithms with channel constraint. We notice 
that an imitation- stable equilibrium is achieved progressively following the dynamics characterized by © 
and ([TTl) . The equilibrium is furthermore very close to the system optimum: we can in fact check that, 
according to Theorem [H the proportion of SUs choosing channel 1, 2 and 3 at the system optimum is 
0.1875, 0.3125 and 0.5 respectively (see also Fig. [4] and Fig. [5]); in the simulation results we observe that 
there are 9, 16 and 25 SUs settling on channel 1, 2 and 3 respectively. 

D. System Fairness 

We now turn to the analysis of the fairness of the proposed spectrum access policies. To this end, we 
adopt the Jain's fairness index [1 IJ, which varies in [0, 1] and reaches its maximum when the resources are 
equally shared amongst users. Fig. [131 whose curves represent an average over 10"^ independent realizations 
of our algorithms, shows that our system turns out to be very fair even from the early iterations. 
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From the same figures one can further infer that indeed DISAP converges more rapidly than PISAP: if 
for instance we fix on the y-axis the fairness value 0.982, the latter is reached by DISAP at the iteration 
t = 100, and by PISAP at the iteration t = 200. 

VI. Related Work 

The spectrum access problem in the considered generic model (with single SU) is closely related to 
the classic Multi-Armed Bandit (MAB) problem lfT2ll . In this case, a SU should strike a balance between 
exploring the environment to find profitable channels (i.e., learn the channel availability probabilities) 
and exploiting the best one using current knowledge. In this line of research, Gittins developed an index 
policy in lfT3l that consists of selecting the arm with the highest index termed as Gittins index. This policy 
is shown to be optimal in the most general case. Lai and Robbins [|T4ll and then Agrawal QS] studied 
the MAB problem by proposing policies based on the upper confidence bounds with logarithmic regret. 
Compared with the classic MAB problem, one major specialty of the spectrum access in cognitive radio 
networks lies in the fact of multiple SUs that can cause collisions if they simultaneously access the same 
channel. Some recent work has investigated this issue, among which Anandkumar et al. proposed two 
algorithms with logarithmic regret, where the number of SUs is known [[T6ll and unknown and estimated 
by each SU [fTTll . Liu and Zhao developed a time-division fare share (TDFS) algorithm with convergence 
and logarithmic regret ifTSl . 

As the spectrum access problem in cognitive radio is essentially a resource allocation problem, another 
important thrust consists of applying game theory to model the competition and cooperation (coordination) 
among SUs and the interaction between SUs and PUs. Along this line of research, a game theoretic 
framework was developed in [19] to analyze the behavior of selfish users in cognitive radio networks, 
resulting in distributed adaptive channel allocation algorithms based on the technique of no-regret learning. 
In Il20l . the convergence of different types of games in cognitive radio systems was studied (i.e., coordinated 
behavior, best response, best response for discounted repeated games, S-modular games and potential 
games). A no-regret learning algorithm was proposed in ||2T]| to address the channel allocation problem in 
cognitive networks. The algorithm can reach a correlated equilibrium which, in many case, is more efficient 
than the classic Nash equilibrium of the game. Maskery et al. considered the dynamic spectrum access 
among cognitive radios from an adaptive, game theoretic learning perspective and proposed decentralized 
dynamic spectrum access protocol [|22ll . Besides, due to the perceived fairness and allocation efficiency, 
auction techniques have also attracted considerable research attention and resulted in a number of auction- 
based spectrum allocation mechanisms (cf. [l23ll and references therein). 

Due to the success of applying evolutionary game theory in the study of biological and economic 
problems, a handful of recent studies have applied evolutionary game theory as a tool to study resource 
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allocation problems arisen from wired and wireless networks, among which Shakkottai et al. addressed 
the problem of non-cooperative multi-homing of users to access points in IEEE 802.11 WLANs by 
modeling it as a population game and studied the equilibrium properties of the game [l24ll : Niyato et 
al. studied the dynamics of network selection in a heterogeneous wireless network using the theory of 
evolutionary game and the replicator dynamic and proposed two network selection algorithm to reach 
the evolutionary equilibrium f25]l; Ackermann et al. investigated the concurrent imitation dynamics in 
the context of symmetric congestion games by focusing on the convergence properties Niyato et al. 
studied the multiple-seller and multiple-buyer spectrum trading game in cognitive radio networks using 
the replicator dynamic and provided a theoretic analysis for the two-seller two-group-buyer case ll26l . 
Coucheney et al. studied the user-network association problem in wireless networks with multi-technology 
and proposed an algorithm to achieve the fair and efficient solution ETl . 

VII. Conclusion and Further Work 

In this paper, we address the spectrum access problem in cognitive radio networks by applying evolu- 
tionary game theory and develop an imitation-based spectrum access policy. We investigate two imitation 
scenarios where a SU can imitate any other SUs and where it can only imitate the other SUs operating on 
the same channel. A systematic theoretical analysis is presented for both scenarios on the induced imitation 
dynamics and the convergence properties of the proposed policy to an imitation- stable equilibrium, which 
is also the e-optimum of the system. As an important direction of the future work, we plan to investigate 
the imitation-based channel access problem in the more generic multi-hop scenario where SUs can imitate 
their neighbors and derive the relevant channel access/assignment policies there. 
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Proof of Theorem ^ 

We first prove the convergence of PISAP to an imitation-stable equilibrium. 

Let i'^"'^ = argmaXjgi^ 7rj(t), z"*'™ = argmin^g^ 7ri(t), we show that for any iteration t, if Hi^ax^t) — 
TTimin{t) > eu, then at least one of the following holds 



I.e., if the difference between the highest expected individual payoff Himax^t) and the worst one 7rjmin(t) 
is larger than eu, we can at least increase ^^i^^n{t) by (^( ^'"'^a'^'^ ) or decrease 7rima.(t) by )■ 

Define the e[/-worst channel set C^^, as the channel set such that a channel i E C^^ iff tTj — Tr^min < eu 
and 7f — TTj > eu/2. In the same way, define the e[/-best channel set C'^'^ as the channel set such that a 
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channel i G C'^' iff iiimax — tti < ejj and tTj — 7f > e[//2. At any iteration t, if TTimax{t) — TTimin{t) > ejj, at 
least one of C^^ and C^^ is not empty. Without loss of generality, assume that C^^ ^ 0. 

For any channel i G C^^{t), let x\{t + 1) denote the proportion of SUs migrating from channel i to / 
after the imitation in the iteration t, i.e., the proportion of SUs operating on channel i at iteration t and 
switching to channel / for iteration t+1. Given that the probability of imitating a SU in channel / is xi{t), 
x\{t + 1) can be computed as 

Xi{t)xi{t)<j[Ki{t) - Tiiit)] IeC~ C,^{t) 

/eajt) 

Denote /S.Xi{t) = + 1) — Xi{t), it follows from the imitation rule that no SU migrates to channel 
^mm_ Hence, Axjmin(t) < and 



x]{t + l] 



^ Xl{t)7ri{t) - ^ Xl{t)7rin.,n{t) 

iec-c^^{t) iec-c^^{t) 



-E[AXim>„(t)] = ^ E[xp"(t + 1)] = Xim.n(t)0- 

i&c~c,^{t) 

= Ximin{t)a Xl{t)[TXl{t) — Tliminit)]. 

Since ^^^(t) 7^ 0, there exists at least a channel /q with vr^y — TTjmm > e^/ and > We have 

— E[AXimin[t)\ > Ximin\t)aXlg{t)['^^l^^[t) — 7rjmm(t)J > — . (14) 

It then follows that 

jjij^min jJjjrnin 



TTjrfLin I 6 ~r -L j TTjrnin 16) 



^ Ar[a;i_„ (t)]2 ^ Ar2a;._„(t) ^ iV2 

Given that tTj G [0, 1], Vi G C and let = minjgc it holds that after at most ^"aeu ^ iterations, 
PISAP converges to an imitation- stable equilibrium where Himax — -n^min < ejj- 

We then show by contradiction that the imitation-stable equilibrium is an e-Nash of G with e = 2eu. 

Denote Xj(oo) and iii{oo) the converged values of Xj and vTj in PISAP. Denote x* = {x*} the NE of 
G. Recall that all payoffs at NE are equal: Hi/Nx* is a constant. Assume now that there exists channel 
iQ such that 

Without loss of generality, assume that 

7ri„(oo) - fiijNxl > lev. 

We have now: 

V? G C, 7ri(oo) - lii/Nx* = vrio(oo) - fii/Nx* + 7ri(oo) - 7ri„(oo) 

= (7rio(oo) - l^iJNx*^) + (7ri(oo) - 7ri,(oo)) 
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> 2eu -eu = eu- 

This leads to Xj(oo) < x*, Vi G C and Ejec^*(°°) = 1 < Z^iec^I ^ which is clearly a contradiction. 



Proof of Theorem \5j[ 

The proof follows the same analysis as that of Theorem [3l With the same notation in the proof of 
Theorem [3j we first show that for each iteration t, if TVimax^t) — 7rjmin(t) > eu, then at least one of the 
following holds 



Tljmin (t + 1) — Tljmin (t) ~ 0( J 

Hlmax{t) — 7lim.ax(t + 1) ~ Q ^ fJ-imaxCrtu 



Under the double imitation, x\{t + 1) can be computed as 

Xi{t)xi{t)pi leC-C,^{t) 
leC,^{t) 

Injecting pi into the above formula, after some algebraic operations and following the same reasoning 
as that in (fT4l) . we have 



-E[AXjmin(t)] 



(J Ob jrnin {t^ 



> 



u — a 
(jj — a N 



uo — a 



^ Xl{t)TXl{t)- ^ Xl{t)^li,.^^,^{t) 

iec-Ce^{t) iec-Ce^{t) 



It then follows that 



jJijrnin 



NXimin{t+\) NXimin(t) N 



Xjmin 



> 



N[Xirmn{t)]'^ (w - tt ) iV%im,n (t ) N'^ 

Given that tTj G [0, 1], Vi G C and let /imm — minjgc /ij, it holds that after at most ^'^atu . 
DISAP converges to an imitation- stable equilibrium where TXimax — Trjm™ < eu. 



Iterations, 



It then can be shown in the same way as in the proof of Theorem [3] that the imitation-stable equilibrium 
is an e-Nash of G with e = 2ej/. ■ 
Proof of theorem ^ 

We prove the statement for t = 2. The case for t > 3 is analogous to (Si, which can be shown by 
induction and is therefore omitted. 

Define the random variable wl (c) such that 

1 if SU c is on channel j at iteration t = 1 

^ii'^) = { and migrates to channel z at t = 2 

otherwise 
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Proportional imitation: If j ^ Sc(l), it holds that w\{c) = 0, otherwise, c imitates a SU that was using 
channel A; at t = and currently (t = 1) on the same channel as c (Sc(l)) with probability and 
migrates to channel i with probability -^*^(o) k- Note that we allow for self-imitation in our algorithm. We 
thus have: 

PK(c) = 1] = <! . 

^r^/^oik otherwise 

fcGC 

We can now derive the population proportions at iteration t = 2 as: 



The expectations of these proportions can now be written as (using the Kronecker delta Sij): 

mm = ^E'Kw = ii = i E "-'"''!'';:nf'""' 



It follows that 



N ^ Uhil) N ^ nJl) ^ xJl 

h,i,kec ^ ^ i,kec ' i,kec 



jec j,i,kec 

As wl (c) and wj (d) are independent random variables for c 7^ c? and since the variance of wl (c) is less 
than 1, the variance of p](2) and Pi{2) for any i,j E C are less than and C/N, respectively. It then 
follows the Bienayme-Chebychev inequality that 

VzGC,P[{b,(2)-Eb,(2)]|>5}]< ^ 



(iV5)2' 

Choosing Nq such that jf^^ < e concludes the proof for t = 2. The proof can then be induced to any 



t as in ||5l . 

Double imitation: If j ^ Sc(l), it holds that wl{c) = 0, otherwise, c imitates two SUs that were using 
respectively channel k and channel z at t = and currently (t = 1) on the same channel as c (Sc(l)) with 
probability jsj"'''^^^ and migrates to channel i with probability -^^^(o) {^2}- 

The proof follows in the steps of the proportional imitation and only the main passages will be sketched 
out. We get: 

K(c) = 1] = <[ ^ '^".(D^^.d) 



k,z€C ^'-^^^ ^"-(1) 
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We then derive the type proportions expectations: 

= 7fEpK(c) = 1] = ^ E 



l,k,z&C 



&N,k,z&C 
l,{k,z}- 



^sc(l)(l)'^s,(l)(l 



"-^sl{o),{fc,z}^i,^c{i) 



It follows that: 



Eb.(2)] 



j6C j,l,k,zeC I JWJ 



The rest of the proof for the double imitation follows the same way as that of proportional imitation. ■ 
Proof of theorem HI' 

We prove the convergence of ^ by showing that the mapping described by Q is a contraction. A 
contraction mapping is defined ll28l as follows: let (X, d) be a metric space, f: X ^ X is a contraction 
if there exists a constant k G [0,1) such that \/x,y E X, d{f{x),f{y)) < kd{x,y), where d{x,y) = 
\\x — y\ \ = maxj |xj — yi\. Such an / is called a contraction and admits a unique fixed point, to which the 
mapping described by / converges. 

Noticing that 



dif{x)Jiy)) = \\f{x)-f{y)\\< 

df 



df 




df 


dx 


■ Ik - y\\ = 


dx 



d{x,y), 



it suffices to show that the Jacobian 



dx 



< k. In our case, it suffices to show that J Lo < k, 



where J = {Jij} is the Jacobian of the mapping described by one of the equation in defined by 

_ dxi{t + l) 

Recall that vTj = and = ^i^, & can be rewritten as 



Ji 



It follows that 



Xi{u) = Xi{u — \) + a 



N 



J i' 



3 







Hence 



\J\ 



max 



otherwise 



j£N I 

which shows that the mapping described by (|9l) is a contraction. It is further easy to check that the fixed 



point of dil) is { 



X 



}, which is also the unique NE of G. 
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Algorithm 1 PISAP: executed at each SU j 

1: Initialization: set the imitation factor a and the imitation threshold eu 

2: For the first iteration t = 1, randomly choose a channel to stay 

3: while at each iteration t > 2 do 

4: Randomly select a SU j' 

5: if Uj < Uji - ejj then 

6: Migrate to the channel Sji with probability p = a{Uj' — Uj) 

1: end if 

8: end while 



Algorithm 2 DISAP: executed at each SU j for each iteration 



1: Initialization: set the two exogenous parameters uj and a such that the payoff of SUs falls into the 

interval set the imitation factor a and the imitation threshold eu 

2: Randomly sample two SUs ji and j2 who, at iteration t — 1, were respectively on channel ii and Z2 



if ii = 


12 then 






Ph -- 


= f [g(f/,j(f/,. 


-u, 






[2- 


Ur-al 
uj—a 1 


Pn -- 


= 






else if 


ii = i then 






Ph -- 


= 






Pn -- 






) + Q(f/,)([/,, 


else 








Ph -- 






) + Qiu,,)iu,, 


P32 -- 


= f [g(f/,j(t/,. 


-U, 




end if 









Uj)]^ where [A]"*" denotes max{0,y4} and 



- u^r 
-m^-Pn 

10: Switch to channel ii with probability pj^ if Uj < Uj^ — eu, switch to channel ^2 with probability pj^ 

if Uj < Uj^ — eu 



Algorithm 3 Imitation-based Spectrum Access Policy under Channel Constraint: executed at each SU j 

1: Initialization: set the imitation factor a and the imitation threshold eu 

2: Randomly choose a channel for the first two iterations t = 0, 1 

3: while for each iteration t > 2 do 

4: Perform imitation in PISAP or DISAP on the same channel 

5: t<-t + l: 

6: end while 
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Fig. 1. Replicators and aggregate monotone dynamics generate a similar phase plane. This is the case of a 2-strategies game with A'^ = 50 
SUs and fi = [0.3 0.8]. As investigated in Section Hill and Section Hll the system has a unique NE, to which all trajectories (solid lines) 
converge exponentially 
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Fig. 11. PISAP: number of SUs per channel as a function of time Fig. 12. DISAP: number of SUs per channel as a function of time 
with channel constraint with channel constraint 
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Fig. 13. Jain's fairness index of the system with channel constraint as a function of time 



